Digital cameras with direct luminance and chrominance detection

ABSTRACT

Digital camera systems and methods are described that provide a color digital camera with direct luminance detection. The luminance signals are obtained directly from a broadband image sensor channel without interpolation of RGB data. The chrominance signals are obtained from one or more additional image sensor channels comprising red and/or blue color band detection capability. The red and blue signals are directly combined with the luminance image sensor channel signals. The digital camera generates and outputs an image in YCrCb color space by directly combining outputs of the broadband, red and blue sensors.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a Continuation of U.S. application Ser. No.16/207,099, filed Dec. 1, 2018, issuing as U.S. Pat. No. 10,694,162 onJun. 23, 2020, which is a Continuation of Ser. No. 15/074,275, filedMar. 18, 2016, now U.S. Pat. No. 10,148,927, which is a Continuation ofU.S. application Ser. No. 14/149,024, filed Jan. 7, 2014, now U.S. Pat.No. 9,294,745, which is a Continuation of U.S. application Ser. No.13/647,708, filed Oct. 9, 2012, now U.S. Pat. No. 8,629,390, which is aContinuation of U.S. patent application Ser. No. 13/100,725, filed May4, 2011, now U.S. Pat. No. 8,304,709, which is a Continuation of U.S.patent application Ser. No. 11/810,623, filed on Jun. 6, 2007, now U.S.Pat. No. 7,964,835, which is a Continuation-In-Part of U.S. patentapplication Ser. No. 11/212,803, filed Aug. 25, 2005, all of which areincorporated herein by reference in their entirety. U.S. applicationSer. No. 11/810,623, claims priority from U.S. Provisional Application60/811,584, filed Jun. 6, 2006, incorporated herein by reference in itsentirety.

TECHNICAL FIELD

The following disclosure relates generally to optical devices and moreparticularly to digital cameras and other systems with direct luminanceand chrominance detection.

BACKGROUND

Color is the perceptual result of light in the visible region of thespectrum (wavelengths approximately in the region of 400 nanometer (nm)to 700 nm) incident upon the retina of the human eye. The human retinahas three types of color photoreceptors or cone cells, which respond toincident radiation with somewhat different spectral response curves.Because there are exactly three types of color photoreceptors, threenumerical components are necessary and sufficient to describe a color,providing that appropriate spectral weighting functions are used. Onedescription of color uses the representation “RGB”, or “RGB colorspace”, and refers to colors red (R), blue (B) and green (G). The red,blue and green colors include the color bands that conventionalsolid-state electronic cameras capture; these colors also approximatelyrepresent colors as viewed by humans. It is a challenge for thedesigners of digital imagers to achieve solutions that provide imagesalmost equivalent to human vision.

Another description of color includes “YUV,” a color encoding systemused for analog television worldwide (NTSC, PAL and SECAM). When colortelevision (TV) signals were developed in the 1950s, YUV was used toencode colors in order to allow black and white TVs to continue toreceive and decode monochrome signals from TV signals, while color setswould decode both monochrome and color signals. The Y in YUV represents“luma” which is brightness, or lightness, and black and white TVs decodeonly the Y part of the signal. The U and V in YUV represent color(chroma) information and are “color difference” signals of blue minusluma (B-Y) and red minus luma (R-Y). The terms luma and chroma are ofteninterchanged with luminance and chrominance, respectively, as thedifference between these terms is a minor difference having to do withuse of gamma corrected or linear pixel signals used in the calculations.

A conventional video camera uses a process referred to as “color spaceconversion” to convert the RGB data captured by its solid-state sensorinto either composite analog signals (YUV) or component versions (analogYPbPr, or digital YCbCr). The difference between YCbCr and RGB is thatYCbCr represents color as brightness and two color difference signals,while RGB represents color as red, green and blue. In YCbCr, the Yrepresents the brightness (luma), Cb represents blue minus luma (B-Y)and Cr represents red minus luma (R-Y). It is desirable in digitalcameras to eliminate RGB conversion and accomplish direct detection ofdigital YCbCr signals within the image sensor. Direct detection of YCbCrwithin the image sensor eliminates the need for RGB conversion, and mayprovide better color rendition and increase image sensor dynamic range.While RGB may be the most commonly used basis for color descriptions, ithas the negative aspect that each of the coordinates (red, green, andblue) is subject to luminance effects from the lighting intensity of theenvironment

Composite analog signals (YUV) (and analog YPbPr or digital YCbCr)reduce transmission bandwidth compared to RGB because the chromachannels (B-Y and R-Y) carry only half the resolution of the luma. YUVis not compressed RGB; rather, Y, B-Y and R-Y are the mathematicalequivalent of RGB. Moving Picture Expert Group (MPEG) compression, whichis used in digital video disks (DVDs or, alternatively, digitalversatile disk), digital TV and video compact disks (CDs), is coded inYCbCr. Furthermore, digital camcorders (e.g., MiniDV, digital video(DV), Digital Betacam, etc.) output YCbCr over a digital link such asFireWire. The reason for using YCrCb signals is that the human eye isless sensitive to chrominance than luminance. Compression algorithms cantake advantage of this phenomenon and subsample the values of Cb and Crwithout significant visual degradation of the original color signal.

Despite improvements in solid-state image sensor and digital cameratechnology, the basic detection mechanism for color cameras is RGB andthe detected signal requires reformatting to YCrCb to separate the RGBsignals into luminance and chrominance data sets for image compressionand resultant image transmission or image data storage. Consequently,there is a need for a digital camera with direct luminance andchrominance detection to eliminate reformatting of RGB signals.

INCORPORATION BY REFERENCE

Each patent, patent application, and/or publication mentioned in thisspecification is herein incorporated by reference in its entirety to thesame extent as if each individual patent, patent application, and/orpublication was specifically and individually indicated to beincorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a conventional digital camera.

FIG. 2 is a block diagram of a digital camera having direct luminanceand chrominance detection, under an embodiment.

FIG. 2A is a flow diagram for directly providing an image in YCrCb colorspace, under an embodiment.

FIG. 3 is a block diagram of a digital camera including four (4) colorcamera channels to directly acquire YCrCb color channel data, under anembodiment.

FIG. 4 is a block diagram of a digital camera including three (3) colorcamera channels to directly acquire YCrCb color channel data, under anembodiment.

FIG. 5 is a block diagram of an example digital camera including two (2)color camera channels to directly acquire YCrCb color channel data,under an embodiment.

FIG. 6 is a block diagram of a digital camera, under an embodiment.

FIG. 7 is an exploded view of a digital camera subsystem, under anembodiment.

FIG. 8 is a block diagram of a digital camera having a three array/lensconfiguration, under an embodiment.

FIG. 9 is a block diagram of a digital camera subsystem that employsseparate arrays on one image sensor, under an embodiment.

FIG. 10 is a block diagram of arrays, each of which receives arespective color as passed by a respective lens, under an embodiment.

FIG. 11 is a block diagram of processing circuitry of a digital camerasubsystem, under an embodiment.

FIG. 12 is a block diagram of signal processing circuitry, under anembodiment.

FIG. 13 is an exploded perspective view of a digital camera, under anembodiment.

FIGS. 14A-14D are schematic exploded representations of one embodimentof an optics portion, under an embodiment.

FIGS. 15A-15C are schematic representations of a sensor array, under anembodiment.

FIG. 16 is a schematic cross-sectional view of a digital cameraapparatus, under an embodiment.

FIG. 17 is a schematic perspective view of a digital camera apparatushaving one or more optics portions with the capability to provide colorseparation, under an embodiment.

FIG. 18A is a block diagram of a processor of a digital camerasubsystem, under an embodiment.

FIG. 18B is a block diagram of a channel processor of a digital camerasubsystem, under an embodiment.

FIG. 18C is a block diagram of an image pipeline of a digital camerasubsystem, under an embodiment.

FIG. 18D is a block diagram of an image post processor of a digitalcamera subsystem, under an embodiment.

FIG. 19 is a block diagram of digital camera system, including systemcontrol components, under an embodiment.

DETAILED DESCRIPTION

The digital camera systems described below provide a color digitalcamera with direct luminance detection. The luminance signals areobtained directly from a broadband image sensor channel withoutinterpolation of RGB data. The chrominance signals are obtained from oneor more additional image sensor channels comprising red and/or bluecolor band detection capability. The red and blue signals are directlycombined with the luminance image sensor channel signals. The digitalcamera generates and outputs an image in YCrCb color space by directlycombining outputs of the broadband, red and blue sensors. The digitalcamera systems also include methods for forming a color image with adigital camera comprising two or more image sensor camera channels, eachchannel having an active area including a plurality of picture elements(pixels) optimized to detect photon wavelengths in bands that directlyrepresent luminance or partial chrominance information.

The digital camera systems of an embodiment provide direct detection ofthe luminance (Y) and chrominance components Cr and Cb of the digitalYCrCb color space widely used in digital imaging and digital imagecompression. These digital camera systems thus eliminate interpolationin the luminance channel for improved color purity, increase sensitivityin the broadband luminance channel for low light performance, optimizeluminance channel design for large dynamic range, and reduce YCrCb colorspace computation. The digital camera systems detect digital YCrCb orgamma corrected YCrCb (Y′CrCb where Y′ is referred to as “luma”).

Digital cameras according to the embodiments described herein includetwo or more closely spaced image sensor camera channels on a commonsubstrate. Each sensor camera channel has its own optics,photo-detection and readout mechanism comprising multiple pictureelements (pixels) with independent signal integration time control. Thepixel area, including photodetector and circuitry, can be as small as 2micrometers (μm) by 2 μm. The individual camera channels look at thesame field of view but are not so limited.

The digital cameras of an embodiment include multiple (e.g. two or more)closely spaced image sensor camera channels including multiple pixels.One camera channel directly provides luminance data in the sense that nopixel interpretation is performed between R, G, and B pixel data in thischannel to generate the luminance data. One or more other camerachannels provide blue and red data as appropriate to a configuration ofthe camera. The combined data from all camera channels provides YCrCbcolor space data for digital imaging, digital compression, digitalstorage and transmission.

In the following description, numerous specific details are introducedto provide a thorough understanding of, and enabling description for,embodiments of the digital camera systems. One skilled in the relevantart, however, will recognize that these embodiments can be practicedwithout one or more of the specific details, or with other components,systems, etc. In other instances, well-known structures or operationsare not shown, or are not described in detail, to avoid obscuringaspects of the disclosed embodiments.

FIG. 1 is a block diagram of a conventional digital camera 100. Thedigital camera 100 generally includes a lens assembly 110, a colorfilter array layer 112, an image sensor 116, an electronic image storagemedia 120, and a power supply 124. The digital camera 100 also includesa peripheral user interface 132 (represented as a shutter button), acircuit board 136 (which supports and electrically interconnects theaforementioned components), a housing 140 (including housing portions141, 142, 143, 144, 145 and 146) and a shutter assembly (not shown),which controls an aperture 150 and passage of light into the digitalcamera 100. A mechanical frame 164 is used to hold the various parts ofthe lens assembly 110 together. The lens assembly 110 includes lenses161, 162 and one or more electro-mechanical devices 163 to move thelenses 161, 162 along a center axis 165. The lenses 161, 162 may be madeup of multiple elements arranged together to form an integral opticalcomponent. Additional lenses may be employed if necessary. Theelectro-mechanical device 163 portion of the lens assembly 110 and themechanical frame 164 portion of the lens assembly 110 may be made up ofnumerous components and/or complex assemblies.

Digital color cameras use either RGB in one pixel, or Bayerrepresentation in which the pixels are arranged in a 2×2 color filterarray pattern where each pixel detects a single color band (R, G or B).The digital camera 100 described above is a camera with a Bayer filterpattern. The color filter array layer 112 has an array of color filtersarranged in a Bayer pattern (e.g., a 2×2 matrix of colors withalternating red and green in one row and alternating green and blue inthe other row, although other colors may be used). The Bayer pattern isrepeated throughout the color filter array.

The image sensor 116 contains a plurality of identical photo detectors(sometimes referred to as “picture elements” or “pixels”) arranged in amatrix. The number of photo detectors is usually in range of hundreds ofthousands to millions. The lens assembly 110 spans the diagonal of thearray.

Each of the color filters in the color filter array 112 is disposedabove a respective one of the photo detectors in the image sensor 116,such that each photo detector in the image sensor receives a specificband of visible light (e.g., red, green or blue) and provides a signalindicative of the color intensity thereof. Conversion of an image fromthis signal format to an RGB format makes use of an interpolation of thetwo missing color values in each pixel. Several standard interpolationmethods (e.g. nearest neighbor, linear, cubic, cubic spline, etc.) canbe used. Signal processing circuitry (not shown) receives signals fromthe photo detectors, processes them (interpolation), and ultimatelyoutputs a color image in RGB and other desired digital formats such asYCrCb.

In conventional interpolation operations, luminance (Y) is calculatedfor each interpolated pixel as

Y=0.299R+0.587G+0.114B.

The two chrominance values (Cr (red chroma) and Cb (blue chroma)) foreach interpolated pixel are calculated as

Cr=0.713(R−Y)=0.500R−0.419G−0.081B, and

Cb=0.564(B−Y)=−0.169R−0.331G+0.500B.

The peripheral user interface 132, which includes the shutter button,may further include one or more additional input devices (e.g., forsettings, controls and/or input of other information), one or moreoutput devices, (e.g., a display for output of images or otherinformation) and associated electronics.

In contrast to the conventional camera that provides luminance datathrough interpolation of R, G, and B pixel data, FIG. 2 is a blockdiagram of a digital camera 200 having direct luminance and chrominancedetection, under an embodiment. The digital camera 200 of an embodimentis configured as a color digital camera with direct luminance detection.The digital camera 200 has multiple camera channels, with some portionof the channels configured for color imaging and some portion of thechannels configured for (broadband) luminance imaging, under anembodiment. The luminance signals are received or obtained from abroadband image sensor channel that is a first set or portion of thecamera channels of the digital camera 200. Chrominance signals arereceived or obtained from one or more additional image sensor channelscontaining red and blue color band detection capability when the red andblue signals are directly combined with the luminance image sensorchannel signals. The additional image sensor channel(s) are a second setor portion of the camera channels of the digital camera 200.

The digital camera 200 includes one or more methods for forming a colorimage with a digital camera comprising two or more image sensor camerachannels. Each of the camera channels includes an active area comprisinga plurality of picture elements (pixels) optimized to detect photonwavelengths in bands that directly represent luminance or partialchrominance information. Generally, the digital camera 200 directlydetects the luminance (Y) and chrominance components (Cr and Cb) of thedigital YCrCb color space widely used in digital imaging and digitalimage compression. The digital camera 200 of an embodiment, by directlydetecting luminance and chrominance information, eliminatesinterpolation in the luminance channel for improved color purity,increased sensitivity in the broadband luminance channel for low lightperformance, optimized luminance channel design for large dynamic range,and reduced YCrCb color space computation. The digital camera 200detects digital YCrCb or gamma-corrected YCrCb (Y′CrCb where Y′ isreferred to as “luma”).

More particularly, FIG. 2A is a flow diagram for directly providing animage in YCrCb color space 290, under an embodiment. The YCbCr colorspace is one in which color is represented as brightness and two colordifference signals. The luminance data represents a photometric measureof the brightness or density of luminous intensity received at thesensor from an image. In YCbCr, therefore, the brightness (luma) isrepresented by Y, and the two color difference signals or componentsinclude a blue chroma component represented by Cb (blue minus luma(B-Y)) and a red chroma component represented by Cr (red minus luma(R-Y)).

In operation, and with reference to FIG. 2A, components of the camera oran embodiment are configured and function to directly detect luminancedata 292 using a broadband wavelength response without the use of RGBcolor data. Chrominance data is generated 294 directly by detecting redcolor band data and blue color band data.

Chrominance is generally represented as two color difference components.Therefore, a red chroma component is generated by directly combining thered color band data with the luminance data (e.g., subtraction operationin which luminance data is subtracted from red color band data).Similarly, a blue chroma component is generated by directly combiningthe blue color band data with the luminance data (e.g., subtractionoperation in which luminance data is subtracted from blue color banddata). The camera components output 296 an image in YCrCb color spaceusing the luminance and chrominance data.

Referring to FIG. 2, the digital camera 200 of an embodiment includestwo or more closely spaced image sensor camera channels on a commonsubstrate. Each sensor camera channel has its own optics,photo-detection and readout mechanism comprising multiple pictureelements (pixels) with independent signal integration time control. Theindividual camera channels look at the same field of view but are not solimited. The closely spaced image sensor camera channels includemultiple pixels. One or more of the camera channels can be used toprovide blue and red data, for example, and a separate camera channelprovides luminance data. The combination of data from all camerachannels provides YCrCb color space data from digital imaging, digitalcompression, digital storage and transmission.

As an example, the digital camera 200 includes a digital camerasubsystem 210, an electronic image storage media 220, a power supply224, and a peripheral user interface 232. The peripheral user interface232 of an embodiment is represented as a shutter button, but is not solimited. The digital camera 200 includes a circuit board 236 which, inan embodiment, supports and/or electrically interconnects one or moreother components of the digital camera 200. The digital camera 200includes a housing 240, including housing portions 241, 242, 243, 244,245 and 246, and a shutter assembly (not shown). The shutter assemblycontrols for example an aperture 250 and passage of light into thedigital camera 200.

The digital camera subsystem 210, also referred to herein as the “DCS”210, includes one or more camera channels. The subsystem 210 of thisexample embodiment includes four camera channels 260A-260D butalternative embodiments are not limited to four camera channels and caninclude any number of camera channels. The DCS 210 of an embodimentreplaces and/or fulfills one, some or all of the roles fulfilled by thelens assembly 110, the color filter 112 and the image sensor 116 of thedigital camera 100 described above with reference to FIG. 1. The fourcamera channels 260A-260D provide four separate color imaging bands. Forexample, channel 260A images the red color band, channel 260B images thegreen color band, channel 260C images the blue color band, and channel260D images the white (broadband) color band. In some imagingapplications, to match the human eye spectral response, channel 260Dwill image wavelengths approximately in the range of 400 nm to 700 nm,while in other imaging applications the spectral band of channel 260Dcan change. For example, the channel 260D can be configured to image ordetect data approximately in a range of 250 nm up to 1060 nm. As anotherexample, the channel 260D can be configured to image or detect dataapproximately in a range of 400 nm up to 1060 nm. Since each camerachannel images in only one color band, pixel interpolation such as thatdescribed for use in digital camera 100 described above is not required.

Camera channel 260D can be used to directly sense and output luminance(Y) values from the image. The camera channel 260D can be configured andfunctions to accommodate the larger signal level associated with thewider broadband wavelength imaging band. The use of a separate camerachannel for obtaining luminance provides increased dynamic range andimproved low light level sensitivity.

Camera channels 260A and 260C collect R and B pixel data, respectively,for use along with the luminance Y data of camera channel 260D ingenerating chrominance (Cr and Cb) information or data as describedabove. Additionally, camera channels 260A, 260B and 260C can be used tocollect R, G and B pixel data respectively, for example. The output ofchannels 260A and 260C can be used to generate chrominance, as well asbeing used along with the output of channel 260B to provide a directoutput of RGB signals from the camera. The RGB signals are output inaddition to the YCrCb signals described herein.

The peripheral user interface 232, which includes the shutter button,may further include one or more additional input devices (e.g., forsettings, controls and/or input of other information), one or moreoutput devices, (e.g., a display for output of images or otherinformation), and associated electronics. The electronic image storagemedia 220, power supply 224, peripheral user interface 232, circuitboard 236, housing 240, shutter assembly (not shown), and aperture 250,may be, for example, similar to the electronic image storage media 120,power supply 124, peripheral user interface 132, circuit board 136,housing 140, shutter assembly (not shown), and aperture 150 of thedigital camera 100 described above.

FIG. 3 is a block diagram of a digital camera subsystem 210, under anembodiment. The DCS 210 includes one or more camera channels (e.g., fourcamera channels 260A-260D) but is not so limited. Each of the camerachannels 260A-260D includes an optics portion and a sensor portion asdescribed above with reference to FIG. 2. The sensor portions of the oneor more camera channels are collectively referred to herein as a sensorsubsystem. Each camera channel has a selectable integration time, andthe integration time setting between camera channels can be different.The integration time adjustment can be used to provide optimum signal tonoise ratio (SNR) and dynamic range in each of the color imaging bandsof the multiple channels (e.g., channels 260A, 260B, 260C and 260D).

The digital camera system 210 further includes a processor. Theprocessor includes an image processor portion 270 or component(hereafter image processor 270) and a controller portion 300 (hereaftercontroller 300 or YCrCb controller 300). The controller portion 300 ispart of the luminance and chrominance signal capability that is obtainedfrom the combined outputs of the different camera channels. Theprocessor 270 is coupled to the one or more sensor portions, e.g.,sensor portions 292A-292D, via one or more communication links,represented by a signal line 330.

A communication link, coupling, or connection may be any kind ofcommunication link including but not limited to, for example, wiredcouplings (e.g., conductors, fiber optic cables), wireless couplings(e.g., acoustic links, electromagnetic links or any combination thereofincluding but not limited to microwave links, satellite links, infraredlinks), and combinations of wired and/or wireless links or couplings.

A description follows of the operation of the DCS of an embodiment. Auser of the host digital camera selects a desired incident light rangeand the camera automatically adjusts the integration time settingbetween the camera channels to give an optimal dynamic range result.Alternatively the camera can automatically adjust integration timecontrol in each channel to provide a desired output signal level lookingat the raw R, G, B and W pixel data and adjusting to a desired signallevel in each channel. The R, G, B and W digital output levels can beadjusted for integration time, dark current offset, responsivity andcolor balance prior to Y, Cr and Cb calculation. The camera can outputYCrCb directly from the W, R and B color channels and RGB directly fromthe R, G and B color channels.

In another embodiment, as shown in FIG. 4, three camera channels areused to directly acquire YCrCb color channel data. Camera channel 260Ais configured for R light, channel 260B is configured for B light, andchannel 260C is configured for W light. Interpolation is not required inan embodiment because each camera channel has only one color pixel, butthe embodiment is not so limited. The integration time of channel 260Ais configured for R band light, the integration time of channel 260B isconfigured for B band light, and the integration time of channel 260C isconfigured for W band light levels. Alternative layout architectures arepossible with the three camera channels as shown in FIG. 4, such as a1.times.3 vertical or a 1×3 horizontal; however the triangular layoutconfiguration, as shown, has area reduction advantages in imager layoutand processing on semiconductor wafers. The triangular layout, as shownin FIG. 4, also provides optical symmetry between the channels. The Rand B pixel data from camera channels 260A and 260B respectively and Ychrominance data from camera channel 260C are used calculate Cr and Cbcomponents directly as described herein.

In another embodiment, as shown in FIG. 5, two camera channels are usedto directly acquire YCrCb color channel data. Camera channel 260A isconfigured for R and B color band detection. The camera channel can usea color filter array approach (50% R and 50% B with alternation colorbands on adjacent pixels) or use a pixel configuration with two-color (Rand B) detection capability within each pixel. In the case of the colorfilter array, pixel interpolation among R and B data is used to achievefull chrominance resolution. A 1×2 vertical layout or 1×2 diagonallayout of camera channels is also possible under an embodiment. Theintegration time of channel 260A is configured for R and B band light,and the integration time of channel 260B is configured for W band lightlevels. The R and B pixel data from camera channels 260A and 260Brespectively and Y chrominance data from camera channel 260C are usedcalculate Cr and Cb components directly.

FIGS. 6-19 illustrate further examples of apparatus and systems in whichthe imaging module and focusing method embodiments disclosed above canbe implemented. FIG. 6 is a block diagram of a digital camera 600, underan embodiment. The digital camera includes a digital camera subsystem602, a circuit board 612, a peripheral user interface electronics 610(here represented as a shutter button, but could also include displayand/or one or more other output devices, setting controls and/or one ormore additional input devices etc), a power supply 606, and electronicimage storage media 604. The digital camera 600 may further include ahousing and a shutter assembly (not shown), which controls an aperture614 and passage of light into the digital camera 600.

FIG. 7 is an exploded view of the digital camera subsystem 602, under anembodiment. In this embodiment, the digital camera subsystem includes animage sensor 704, an optics frame (also referred to as a frame) 702, andlenses 712A-712D. The frame 702 is used to mount the lenses 712A-712D tothe image sensor 704. The image sensor, or imager die 704 generallyincludes a semiconductor integrated circuit or “chip” having severalhigher order features including multiple arrays 704A-704D and signalprocessing circuits 708 and 710. Each of the arrays 704A-704D capturesphotons and outputs electronic signals. The signal processing circuit708, in certain embodiments, processes signals for each of theindividual arrays 704. The signal processing circuit 710 may combine theoutput from signal processing 708 into output data (usually in the formof a recombined full color image). Each array and the related signalprocessing circuitry may be tailored to address a specific band ofvisible spectrum. The imaging sensor 704 shows a singleanalog-to-digital conversion (ADC) however separate ADCs can be used foreach channel located at the array output, array column signal outputs orwithin each pixel of the arrays 704.

Each of lenses 712A-712D may be tailored for the respective wavelengthof the respective array. Lenses are approximately the same size as theunderlying array 704, and will differ from one another in size and shapedepending upon the dimensions of the underlying array and the wavelengththe array is configured to receive. In alternative embodiments a lenscould cover only a portion of an array, and could extend beyond thearray. Lenses can comprise any suitable material or materials, includingfor example, glass and plastic. Lenses can be doped in any suitablemanner, such as to impart a color filtering, polarization, or otherproperty. Lenses can be rigid or flexible.

In the example of FIG. 7, each lens, array, and signal processingcircuit constitutes an image generating subsystem for a band of visiblespectrum (e.g., red, blue, green, etc). These individual images are thencombined with additional signal processing circuitry within thesemiconductor chip to form a full image for output.

Although the digital camera subsystem 704 is depicted in a fourarray/lens configuration, the digital camera subsystem can be employedin a configuration having any number of arrays/lenses and anycombination of shapes of arrays/lenses. FIG. 8 is a block diagram of adigital camera 800 having a three array/lens configuration, under anembodiment. The digital camera 800 includes a digital camera subsystem802 that includes three lenses. The digital camera 800 further includesa circuit board 812, a peripheral user interface electronics 810 (hererepresented as a shutter button, but could also include display and/orone or more other output devices, setting controls and/or one or moreadditional input devices etc), a power supply 806, and electronic imagestorage media 804. The digital camera 800 may further include a housingand a shutter assembly (not shown), which controls an aperture 814 andpassage of light into the digital camera 800.

FIG. 9 is a block diagram of a digital camera subsystem that employsseparate arrays, e.g., arrays 904A-904D, on one image sensor, incontrast to the prior art. For example, typical prior art approachesemploy a Bayer pattern (or variations thereof), perform operationsacross the array (a pixel at a time), and integrate each set of fourpixels (for example, red/green/blue/green or variation thereof) from thearray into a single full color pixel.

Each of the arrays 904 focuses on a specific band of visible spectrum.Each lens only needs to pass a respective color (906A-906D) on to theimage sensor. The traditional color filter sheet is eliminated. Eacharray 904 outputs signals to signal processing circuitry. Signalprocessing circuitry for each of these arrays is also tailored for eachof the bands of visible spectrum. In effect, individual images arecreated for each of these arrays. Following this process, the individualimages are combined or to form one full color or black/white image. Bytailoring each array and the associated signal processing circuitry, ahigher quality image can be generated than the image resulting fromtraditional image sensors of like pixel count.

As such, each array may be configured or optimized to be more efficientin capturing and processing the image in that particular color.Individual lenses (912A-D) can be tailored for the array's band of colorspectrum.

FIG. 10 is a block diagram of arrays 1004A-1004D. Each array 1004receives a respective color as passed by a respective lens. Thetraditional color filter sheet is eliminated. Each array 1004 outputssignals to signal processing circuitry. Signal processing circuitry foreach of these arrays is also tailored for each of the bands of visiblespectrum. In effect, individual images are created for each of thesearrays. Following this process, the individual images are combined or toform one full color or black/white image. By tailoring each array andthe associated signal processing circuitry, a higher quality image canbe generated than the image resulting from traditional image sensors oflike pixel count.

FIG. 11 is a block diagram of processing circuitry of a digital camerasubsystem, under an embodiment. FIG. 11 includes an array 1104,including arrays 1104A-1104D, and signal processing circuitry (alsoreferred to as image processing circuitry) 1214 and 1216. Each arrayoutputs signals to signal processing circuitry.

FIG. 12 is a block diagram of image processing circuitry 1214 and 1216.Within the image processing circuitry 1214, each array can be processedseparately to tailor the processing to the respective bands of spectrum.

Column logic 1214.1A-1214.1D is the portion of the signal processingcircuitry that reads the signals from the pixels. For example, thecolumn logic 1214.1A reads signals from the pixels in array 1204A.Column logic 1214.1B reads signals from the pixels in array 1204B.Column logic 1214.1C reads signals from the pixels in array 1204C.Column logic 1214.1D reads signals from the pixels in array 1204D.

Since an array is targeting a specific wavelength, wavelengths, band ofwavelength, or band of wavelengths, the column logic may have differentintegration times for each array enhancing dynamic range and/or colorspecificity. Signal processing circuitry complexity for each array canbe substantially reduced since logic may not have to switch betweenextreme color shifts.

Analog Signal Logic (ASL) 1214.2A-1214.2D for each array may be colorspecific. As such, the ASL processes a single color and therefore can beoptimized for gain, noise, dynamic range, linearity, etc. Due to colorsignal separation, dramatic shifts in the logic and settling time arenot required as the amplifiers and logic do not change on a pixel bypixel (color to color) basis as in traditional Bayer patterned designs.Alternatively, digital logic may be used instead of or in combinationwith the ASL (e.g., arrays including ADCs at the column level). In someconfigurations, for example, where the ADC is external to the columnthen digital signal logic would be used with digital signal processingfrom this point onwards in the processing chain.

Black level control 1214.3A-1214.3D assesses the level of noise withinthe signal, and filters it out. With each array focused upon a narrowerband of visible spectrum than traditional image sensors, the black levelcontrol can be more finely tuned to eliminate noise.

Exposure control 1214.4A-1214.4D measures the overall volume of lightbeing captured by the array and adjusts the capture time for imagequality. Traditional cameras must make this determination on a globalbasis (for all colors). The embodiments describe herein allow forexposure control to occur differently for each array and targeted bandof wavelengths.

These processed images are then passed to a second group of signalprocessing circuitry 1216. First, image processing logic 1216.1integrates the multiple color planes into a single color image. Theimage is adjusted for saturation, sharpness, intensity, hue, artifactremoval, and defective pixel correction.

In an embodiment, the final two operations include encoding the signalinto standard protocols such as MPEG, JPEG, etc. in an encoder 1216.2before passing the result to a standard output interface 1216.3, such asUSB.

Although the signal processing circuitries 1214 and 1216 are shown atspecific areas of the image sensor, the signal processing circuitries1214 and 1216 can be placed anywhere on the chip and subdivided in anyfashion. The signal processing circuitries are often placed in multiplelocations. While some signal processing can be accomplished in analogformat, signals can be digitized and signal processing accomplished indigital format.

As previously stated, the image sensor 1204 generally includes asemiconductor chip having several higher order features includingmultiple arrays (1204A-1204D), and signal processing circuitry 1214, inwhich each array and the related signal processing circuitry ispreferably tailored to address a specific band of visible spectrum. Asnoted above, the image sensor array can be configured using any multiplenumbers and shapes of arrays.

The image sensor 1204 can be constructed using any suitable technology,including silicon and germanium technologies. The pixels can be formedin any suitable manner, can be sized and dimensioned as desired, and canbe distributed in any desired pattern. Pixels that are distributedwithout any regular pattern may also be used.

Any range of visible spectrum can be applied to each array depending onthe specific interest of the customer or application. Further, aninfrared array could also be employed as one of the array/lenscombinations giving low light capabilities to the sensor.

As previously described, arrays 1204A-1204D may be of any size or shape.While some figures referenced herein show the arrays as individual,discrete sections of the image sensor, these arrays may also betouching. There may also be one large array configured such that thearray is subdivided into sections, and each section is focused upon oneband of spectrum, creating the same effect as separate arrays on thesame chip.

Although the well depth (photon collection depth) of the photo detectorsacross each individual array 1204 may be the same, the well depth of anygiven array may be different from that of other arrays of the sensorsubsystem. A photo detector includes an area or portion of the photodetector that captures, collects, is responsive to, detects and/orsenses the intensity illumination of incident light. In someembodiments, the well depth starts at the surface of the photo detectorand proceeds into a doped semiconductor region, in other embodiments thewell depth is located within a buried region of the semiconductor.

Selection of an appropriate well depth depends on many factors,including the targeted band of visible spectrum. Since each entire arrayis likely to be targeted at one band of visible spectrum (e.g., red) thewell depth can be configured to capture that wavelength and ignoreothers (e.g., blue, green). Doping of the semiconductor material in thecolor specific arrays can further be used to enhance the selectivity ofthe photon absorption for color-specific wavelengths.

In various embodiments, a digital camera subsystem can have multipleseparate arrays on a single image sensor, each with its own lens. Thesimple geometry of smaller, multiple arrays allows for a smaller lenses(e.g., smaller diameter, thickness and focal length), which allows forreduced stack height in the digital camera.

The lens and frame concept is applicable to traditional image sensors(without the traditional color filter sheet) to gain physical size, costand performance advantages.

Each array can advantageously be focused on one band of visible and/ordetectable spectrum. Among other things, each lens may be tuned forpassage of one specific band of wavelength. Since each lens wouldtherefore not need to pass the entire light spectrum, the number ofelements may be reduced, for example, to one or two from three or morelenses.

Further, due to the focused bandwidth for each lens, each of the lensesmay be dyed (doped) during the manufacturing process for its respectivebandwidth (e.g., red for the array targeting the red band of visiblespectrum). Alternatively, a single color filter may be applied acrosseach lens. This process eliminates the traditional color filters (suchas the sheet of individual pixel filters) thereby reducing cost,improving signal strength and eliminating the pixel reduction barrier.

The above-described devices can include any suitable number ofcombinations, including as few as two arrays/lenses, and many more thantwo arrays/lenses. Examples include: two arrays/lenses configured asred/green and blue; two arrays/lenses configured as red and blue/green;two arrays/lenses configured as red, green, blue; four arrays/lensesconfigured as red, blue, green, emerald (for color enhancement); fourarrays/lenses configured as red, blue, green, infrared (for low lightconditions); and eight arrays/lenses configured as double the aboveconfigurations for additional pixel count and image quality.

The cameras or camera subsystems described herein are intended to beemblematic of a generic appliance containing the digital camerasubsystem. Thus, the description herein should be interpreted as beingemblematic of still and video cameras, cell phones, other personalcommunications devices, surveillance equipment, automotive applications,computers, manufacturing and inspection devices, toys, plus a wide rangeof other and continuously expanding applications. Of course thesealternative interpretations may or may not include the specificcomponents as depicted herein. For example, the circuit board may not beunique to the camera function but rather the digital camera subsystemmay be an add-on to an existing circuit board, such as in a cell phone.

Any or all of the methods and/or apparatus disclosed herein may beemployed in any type of apparatus or process including, but not limitedto still and video cameras, cell phones, other personal communicationsdevices, surveillance equipment, automotive applications, computers,manufacturing and inspection devices, toys, plus a wide range of otherand continuously expanding applications.

Although each array and the related signal processing circuitry is canbe tailored to address a specific band of visible spectrum, and eachlens may be tuned for passage of that one specific band of wavelength,there is no requirement that each such array and the related signalprocessing circuitry be tailored to address a specific band of thevisible spectrum. Nor is there any requirement that each lens be tunedfor passage of a specific band of wavelength or that each of the arraysbe located on the same semiconductor device. Indeed, the embodimentsdescribed and illustrated herein, including the specific componentsthereof, need not employ wavelength-specific features. For example, thearrays and/or signal processing circuitry need not be tailored toaddress a specific wavelength or band of wavelengths.

FIG. 13 is an exploded perspective view of a digital camera 1300, underan embodiment. The digital camera apparatus 1300 includes one or moresensor arrays, e.g., four sensor arrays 1304A-1304D, and one or moreoptics portions, e.g., four optics portions 1312A-1312D. Each of theoptics portions 1304A-1304D may include a lens, and may be associatedwith a respective one of the sensor arrays sensor arrays 1304A-1304D. Insome embodiments a support 1302, for example a frame, is provided tosupport the one or more optics portions 1312A-1312D, at least in part.Each sensor array and the respective optics portion may define anoptical channel. For example, an optical channel 1306A may be defined bythe optics portion 1312A and the sensor array 1304A. An optical channel1306B may be defined by the optics portion 1312B and the sensor array1304B. An optical channel 1306C may be defined by optics portion 1312Cand the sensor array 1304C. An optical channel 1306D may be defined byoptics portion 1312D and a sensor array 1304D. The optics portions ofthe one or more optical channels are also collectively referred to as anoptics subsystem.

The sensor arrays of the one or more optical channels are collectivelyreferred as a sensor subsystem. The two or more sensor arrays may beintegrated in or disposed on a common substrate, referred to as an imagedevice, on separate substrates, or any combination thereof. For example,where the system includes three or more sensor arrays, two or moresensor arrays may be integrated in a first substrate, and one or moreother sensor arrays may be integrated in or disposed on a secondsubstrate.

In that regard, the one or more sensor arrays 1304A-1304D, may or maynot be disposed on a common substrate. For example, in some embodimentstwo or more of the sensor arrays are disposed on a common substrate. Insome embodiments, however, one or more of the sensor arrays is notdisposed on the same substrate as one or more of the other sensorarrays. The one or more optical channels may or may not be identical toone another.

In some embodiments, one of the optical channels 1306 detects red light,one of the optical channels 1306 detects green light, and one of theoptical channels 1306 detects blue light. In some of such embodiments,one of the optical channels 1306 detects infrared light, cyan light, oremerald light. In some other embodiments, one of the optical channels1306 detects cyan light, one of the optical channels 1306 detects yellowlight, one of the optical channels 1306 detects magenta light and one ofthe optical channels 1306 detects clear light (black and white). Anyother wavelength or band of wavelengths (whether visible or invisible)combinations can also be used.

A processor 1314 is coupled to the one or more sensor arrays1304A-1304D, via one or more communication links, e.g., communicationlinks 1308A-1308D, respectively. A communication link may be any kind ofcommunication link including but not limited to, for example, wired(e.g., conductors, fiber optic cables) or wireless (e.g., acousticlinks, electromagnetic links or any combination thereof including butnot limited to microwave links, satellite links, infrared links), andcombinations thereof, each of which may be public or private, dedicatedand/or shared (e.g., a network). A communication link may include forexample circuit switching or packet switching or combinations thereof.Other examples of communication links include dedicated point-to-pointsystems, wired networks, and cellular telephone systems. A communicationlink may employ any protocol or combination of protocols including butnot limited to the Internet Protocol.

The communication link may transmit any type of information. Theinformation may have any form, including, for example, but not limitedto, analog and/or digital) e.g., a sequence of binary values, or a bitstring). The information may or may not be divided into blocks. Ifdivided into blocks, the amount of information in a block may bepredetermined or determined dynamically, and/or may be fixed (e.g.,uniform) or variable.

As will be further described hereinafter, the processor may include oneor more channel processors, each of which is coupled to a respective one(or more) of the optical channels and generates an image based at leastin part on the signal(s) received from the respective optical channel,although this is not required. In some embodiments, one or more of thechannel processors is tailored to its respective optical channel, forexample, as described herein. For example, when one of the opticalchannels is dedicated to a specific wavelength or color (or band ofwavelengths or colors) the respective channel processor may be adaptedor tailored to such wavelength or color (or band of wavelengths orcolors). Further, the gain, noise reduction, dynamic range, linearityand/or any other characteristic of the processor, or combinations ofsuch characteristics, may be adapted to improve and/or optimize theprocessor to such wavelength or color (or band of wavelengths orcolors). Tailoring the channel processing to the respective opticalchannel may facilitate generating an image of a quality that is higherthan the quality of images resulting from traditional image sensors oflike pixel count. In addition, providing each optical channel with adedicated channel processor may help to reduce or simplify the amount oflogic in the channel processors as the channel processor may not need toaccommodate extreme shifts in color or wavelength, e.g., from a color(or band of colors) or wavelength (or band of wavelengths) at oneextreme to a color (or band of colors) or wavelength (or band ofwavelengths) at another extreme.

In operation, an optics portion of a optical channel receives light fromwithin a field of view and transmits one or more portions of such light,e.g., in the form of an image at an image plane. The sensor arrayreceives one or more portions of the light transmitted by the opticsportion and provides one or more output signals indicative thereof. Theone or more output signals from the sensor array are supplied to theprocessor. In some embodiments, the processor generates one or moreoutput signals based, at least in part, on the one or more signals fromthe sensor array. In some other embodiments, the processor may generatea combined image based, at least in part, on the images from two or moreof such optical channels.

Although the processor 1314 is shown separate from the one or moresensor arrays 1304A-1304D, the processor 1314, or portions thereof, mayhave any configuration and may be disposed in one or more locations. Forexample, certain operations of the processor may be distributed to orperformed by circuitry that is integrated in or disposed on the samesubstrate or substrates as one or more of the one or more of the sensorarrays and certain operations of the processor are distributed to orperformed by circuitry that is integrated in or disposed on one or moresubstrates that are different from (whether such one or more differentsubstrates are physically located within the camera or not) thesubstrates the one or more of the sensor arrays are integrated in ordisposed on.

The digital camera apparatus 1300 may or may not include a shutter, aflash and/or a frame to hold the components together.

FIGS. 14A-14D are schematic exploded representations of one embodimentof an optics portion, such as optic portion 1312A, under an embodiment.In FIG. 14A, the optics portion 1312A includes one or more lenses, e.g.,a complex lens module 1480, one or more color coatings, e.g., a colorcoating 1482, one or more masks, e.g., an auto focus mask 1484, and oneor more IR coatings, e.g., an IR coating 1486.

Lenses can comprise any suitable material or materials, including forexample, glass and plastic. Lenses can be doped (dyed) or manufacturedin any suitable manner, such as to impart a color filtering,polarization, or other property. Lenses can be rigid or flexible. Inthis regard, some embodiments employ a lens (or lenses) having a dyecoating, a dye diffused in an optical medium (e.g., a lens or lenses), asubstantially uniform color filter and/or any other filtering techniquethrough which light passes to the underlying array.

The color coating 1482 helps the optics portion filter (or substantiallyattenuate) one or more wavelengths or bands of wavelengths. The autofocus layer 1484 may define one or more interference patterns that helpthe digital camera apparatus perform one or more auto focus functions.The IR coating 1486 helps the optics portion 1312A filter a wavelengthor band of wavelength in the IR portion of the spectrum.

The one or more color coatings, e.g., color coating 1482, one or moremasks, e.g., mask 1484, and one or more IR coatings, e.g., IR coating1486 may have any size, shape and/or configuration.

In some embodiments, as shown in FIG. 14B, one or more of the one ormore color coatings, e.g., the color coating 1482, are disposed at thetop of the optics portion. Some embodiments of the optics portion(and/or components thereof) may or may not include the one or more colorcoatings, one or more masks and one or more IR coatings and may or maynot include features in addition thereto or in place thereof.

In some embodiments, as shown in FIG. 14C, one or more of the one ormore color coatings, e.g., the color coating 1482, are replaced by oneor more filters 1488 disposed in the optics portion, e.g., disposedbelow the lens. In other embodiments, as shown in FIG. 14D, one or moreof the color coatings are replaced by one or more dyes diffused in thelens.

The one or more optics portions, e.g., optics portions 1312A-1312D, mayor may not be identical to one another. In some embodiments, forexample, the optics portions are identical to one another. In some otherembodiments, one or more of the optics portions are different, in one ormore respects, from one or more of the other optics portions. Forexample, in some embodiments, one or more of the characteristics (forexample, but not limited to, its type of element(s), size, response,and/or performance) of one or more of the optics portions is tailored tothe respective sensor array and/or to help achieve a desired result. Forexample, if a particular optical channel is dedicated to a particularcolor (or band of colors) or wavelength (or band of wavelengths) thenthe optics portion for that optical channel may be adapted to transmitonly that particular color (or band of colors) or wavelength (or band ofwavelengths) to the sensor array of the particular optical channeland/or to filter out one or more other colors or wavelengths. In some ofsuch embodiments, the design of an optical portion is optimized for therespective wavelength or bands of wavelengths to which the respectiveoptical channel is dedicated. It should be understood, however, that anyother configurations may also be employed. Each of the one or moreoptics portions may have any configuration.

In some embodiments, each of the optics portions, e.g., optics portions1312A-1312D of FIG. 13, comprises a single lens element or a stack oflens elements (or lenslets), although, as stated above. For example, insome embodiments, a single lens element, multiple lens elements and/orcompound lenses, with or without one or more filters, prisms and/ormasks are employed.

An optical portion can also contain other optical features that aredesired for digital camera functionality and/or performance. Forexample, these features can include electronically tunable filters,polarizers, wavefront coding, spatial filters (masks), and otherfeatures not yet anticipated. Some of the features (in addition to thelenses) are electrically operated (such as a tunable filter), or aremechanically movable with MEMs mechanisms.

In some embodiments, one or more photochromic (or photochromatic)materials are employed in one or more of the optical portions. The oneor more materials may be incorporated into an optical lens element or asanother feature in the optical path, for example, above one or more ofthe sensor arrays. In some embodiments, photochromatic materials may beincorporated into a cover glass at the camera entrance (common aperture)to all optics (common to all optical channels), or put into the lensesof one or more optical channels, or into one or more of the otheroptical features included into the optical path of an optics portionover any sensor array.

FIGS. 15A-15C are schematic representations of one embodiment of asensor array 1504. The sensor array is similar to one of the sensorarrays 1304A-1304D of FIG. 13, for example. As shown in FIG. 15A, thesensor array 1504 is coupled to circuits 1570, 1572, and 1574. Thesensor array sensor array 1504 captures light and converts it into oneor more signals, such as electrical signals, which are supplied to oneor more of the circuits 1570, 1572, and 1574. The sensor array 1504includes a plurality of sensor elements such as for example, a pluralityof identical photo detectors (sometimes referred to as “pictureelements” or “pixels”), e.g., pixels 1580 _(1,1)-1580 _(n,m). The photodetectors 1580 _(1,1)-1580 _(n,m), are arranged in an array, for examplea matrix-type array. The number of pixels in the array may be, forexample, in a range from hundreds of thousands to millions. The pixelsmay be arranged for example, in a two-dimensional array configuration,for example, having a plurality of rows and a plurality of columns,e.g., 640×480, 1280×1024, etc. However, the pixels can be sized anddimensioned as desired, and can be distributed in any desired pattern.Pixels that are distributed without any regular pattern can also used.Referring to FIG. 15B, a pixel, for example pixel 1580 _(1,1), may beviewed as having x and y dimensions, although the photon capturingportion of a pixel may or may not occupy the entire area of the pixeland may or may not have a regular shape. In some embodiments, the sensorelements are disposed in a plane, referred to herein as a sensor plane.The sensor may have orthogonal sensor reference axes, including forexample, an x-axis, a y-axis, and a z-axis, and may be configured so asto have the sensor plane parallel to the x-y plane XY and directedtoward the optics portion of the optical channel. Each optical channelhas a field of view corresponding to an expanse viewable by the sensorarray. Each of the sensor elements may be associated with a respectiveportion of the field of view.

The sensor array may employ any type of technology, for example, but notlimited to MOS pixel technologies (e.g., one or more portions of thesensor are implemented in “Metal Oxide Semiconductor” technology),charge coupled device (CCD) pixel technologies, or combination of both.The sensor array may comprise any suitable material or materials,including, but not limited to, silicon, germanium and/or combinationsthereof. The sensor elements or pixels may be formed in any suitablemanner.

In operation, the sensor array 1504A, is exposed to light on asequential line per line basis (similar to a scanner, for example) orglobally (similar to conventional film camera exposure, for example).After being exposed to light for certain period of time (exposure time),the pixels 1580 _(1,1)-1580 _(n,m), are read out, e.g., on a sequentialline per line basis.

In some embodiments, circuitry 1570, also referred to as column logic1570, is used to read the signals from the pixels 1580 _(1,1)-1580_(n,m). FIG. 15C is a schematic representation of a pixel circuit. Thepixels 1580 _(1,1)-1580 _(n), also referred to as sensor elements, maybe accessed one row at a time by asserting one of the word lines 1582,which run horizontally through the sensor array 1504A. A single pixel1580 _(1,1) is shown. Data is passed into and/or out of the pixel 1580_(1,1) via bit lines (such as bit line 1584) which run verticallythrough the sensor array 1504A.

The pixels are not limited to the configurations shown in FIGS. 15A-15C.As stated above, each of the one or more sensor arrays may have anyconfiguration (e.g., size, shape, pixel design, and pixel electroniccircuitry).

The sensor arrays 1302A-1302D of FIG. 13 may or may not be identical toone another. In some embodiments, for example, the sensor arrays areidentical to one another. In some other embodiments, one or more of thesensor arrays are different, in one or more respects, from one or moreof the other sensor arrays. For example, in some embodiments, one ormore of the characteristics (for example, but not limited to, its typeof element(s), size (for example, surface area), and/or performance) ofone or more of the sensor arrays is tailored to the respective opticsportion and/or to help achieve a desired result.

FIG. 16 is a schematic cross-sectional view of a digital cameraapparatus 1600 including a printed circuit board 1620 of a digitalcamera on which the digital camera elements are mounted, under anembodiment. In this embodiment, the one or more optics portions, e.g.,optics portions 1612A and 1612B are seated in and/or affixed to asupport 1614. The support 1614 (for example a frame) is disposedsuperjacent a first bond layer 1622, which is disposed superjacent animage device 1620, in or on which sensor portions 1612A-1612D (sensorportions 1612C and 1612D are not shown), are disposed and/or integrated.The image device 1620 is disposed superjacent a second bond layer 1624which is disposed superjacent the printed circuit board 1621.

The printed circuit board 1621 includes a major outer surface 1630 thatdefines a mounting region on which the image device 1620 is mounted. Themajor outer surface 1630 may further define and one or more additionalmounting regions (not shown) on which one or more additional devicesused in the digital camera may be mounted. One or more pads 1632 areprovided on the major outer surface 1630 of the printed circuit board toconnect to one or more of the devices mounted thereon.

The image device 1620 includes the one or more sensor arrays (notshown), and one or more electrically conductive layers. In someembodiments, the image device 1620 further includes one, some or allportions of a processor for the digital camera apparatus 1600. The imagedevice 1620 further includes a major outer surface 1640 that defines amounting region on which the support 1614 is mounted.

The one or more electrically conductive layers may be patterned todefine one or more pads 1642 and one or more traces (not shown) thatconnect the one or more pads to one or more of the one or more sensorarrays. The pads 1642 are disposed, for example, in the vicinity of theperimeter of the image device 1620, for example along one, two, three orfour sides of the image device 1620. The one or more conductive layersmay comprise, for example, copper, aluminum, and/or any other suitablyconductive material(s).

A plurality of electrical conductors 1650 may connect one or more of thepads 1642 on the image device 1620 to one or more of the pads 1632 onthe circuit board 1621. The conductors 1650 may be used, for example, toconnect one or more circuits on the image device 1620 to one or morecircuits on the printed circuit board 1621.

The first and second bond layers 1622 and 1624 may comprise any suitablematerial(s), including but not limited to adhesive, and may comprise anysuitable configuration. The first and second bond layers 1622, 1624 maycomprise the same material(s) although this is not required. As usedherein, a bond layer may be continuous or discontinuous. For example, aconductive layer may be an etched printed circuit layer. Moreover, abond layer may or may not be planar or even substantially planar. Forexample, a conformal bond layer on a non-planar surface will benon-planar. While a wire bonded attachment is shown in FIG. 16,electrical vias can be made through image device 120 to provide a bumpinterconnect to pads 1632 on circuit board 1621.

FIG. 17 is a schematic perspective view of a digital camera apparatushaving one or more optics portions with the capability to provide colorseparation in accordance with one embodiment of the present disclosure.In some of such embodiments, one or more of the optics portions, e.g.,optics portion 1712C includes an array of color filters (e.g., Bayerpattern, Bayer pattern on the imaging array beneath the optics portion,a color filter on the imaging array and separate from the lensassembly). In some of such embodiments, one or more of the opticsportions, e.g., optics portion 1712C has the capability to provide colorseparation similar to that which is provided by a color filter array.

In some embodiments, the lens and/or filter of the optical channel maytransmit both of such colors or bands of colors, and the optical channelmay include one or more mechanisms elsewhere in the optical channel toseparate the two colors or two bands of colors. For example, a colorfilter array may be disposed between the lens and the sensor array,and/or the optical channel may employ a sensor capable of separating thecolors or bands of colors. In some of the latter embodiments, the sensorarray may be provided with pixels that have multiband capability, e.g.,two or three colors. For example, each pixel may comprise two or threephotodiodes, wherein a first photodiode is adapted to detect a firstcolor or first band of colors, a second photodiode is adapted to detecta second color or band of colors and a third photodiode is adapted todetect a third color or band of colors. One way to accomplish this is toprovide the photodiodes with different structures and/or characteristicsthat make them selective, such that the first photodiode has a highersensitivity to the first color or first band of colors than to thesecond color or band of colors, and the second photodiode has a highersensitivity to the second color or second band of colors than to thefirst color or first band of colors. Alternatively, the photodiodes aredisposed at different depths in the pixel, taking advantage of thedifferent penetration and absorption characteristics of the differentcolors or bands of colors. For example, blue and blue bands of colorspenetrate less (and are thus absorbed at a lesser depth) than green andgreen bands of colors, which in turn penetrate less (and are thusabsorbed at a lesser depth) than red and red bands of colors. In someembodiments, such a sensor array is employed, even though the pixels maysee only one particular color or band of colors, for example, to inorder to adapt such sensor array to the particular color or band ofcolors.

FIG. 18A is a block diagram of a processor 1802 of a digital camerasubsystem 1800, under an embodiment. In this embodiment, the processor1802 includes one or more channel processors, one or more imagepipelines, and/or one or more image post processors. Each of the channelprocessors is coupled to a respective one of the optical channels (notshown) and generates an image based at least in part on the signal(s)received from the respective optical channel. In some embodiments theprocessor 1802 generates a combined imaged based at least in part on theimages from two or more of the optical channels. In some embodiments,one or more of the channel processors are tailored to its respectiveoptical channel, as previously described.

In various embodiments, the gain, noise reduction, dynamic range,linearity and/or any other characteristic of the processor, orcombinations of such characteristics, may be adapted to improve and/oroptimize the processor to a wavelength or color (or band of wavelengthsor colors). Tailoring the channel processing to the respective opticalchannel makes it possible to generate an image of a quality that ishigher than the quality of images resulting from traditional imagesensors of like pixel count. In such embodiments, providing each opticalchannel with a dedicated channel processor helps to reduce or simplifythe amount of logic in the channel processors, as the channel processormay not need to accommodate extreme shifts in color or wavelength, e.g.,from a color (or band of colors) or wavelength (or band of wavelengths)at one extreme to a color (or band of colors) or wavelength (or band ofwavelengths) at another extreme

The images (and/or data which is representative thereof) generated bythe channel processors are supplied to the image pipeline, which maycombine the images to form a full color or black/white image. The outputof the image pipeline is supplied to the post processor, which generatesoutput data in accordance with one or more output formats.

FIG. 18B shows one embodiment of a channel processor. In thisembodiment, the channel processor includes column logic, analog signallogic, and black level control and exposure control. The column logic iscoupled to the sensor and reads the signals from the pixels. Each of thecolumn logic, analog signal logic, digital signal logic, black levelcontrol and exposure control can be configured for processing asappropriate to the corresponding optical channel configuration (e.g.,specific wavelength or color, etc.). For example, the analog signallogic is optimized, if desired, for processing. Therefore, gain, noise,dynamic range and/or linearity, etc., are optimized as appropriate tothe corresponding optical channel configuration (e.g., a specificwavelength or color, etc.). As another example, the column logic mayemploy an integration time or integration times adapted to provide aparticular dynamic range as appropriate to the corresponding opticalchannel.

The digital camera systems of an embodiment provide digital cameras withlarge effective single-frame dynamic exposure ranges through the use ofmultiple camera channels, including multiple optics and image sensors.The multiple camera channels are all configured to image the same fieldof view simultaneously, and each operates independently under adifferent integration time. The digital camera can include, for example,a 3×3 assembly of image sensors, perhaps three sensor of each color(e.g., red (R), green (G), and blue (B)) and the integration time of thesensors associated with each color can be varied, for example, eachcolor can have three distinct values (e.g., 0.1 msec, 1 msec, and 10msec integration time, respectively). The data from all sensors can bedigitally combined to provide a much greater dynamic range within oneframe of digital camera data. The raw digital camera data could be usedby digital signal processing of the scene. The digital data can also bestored and displayed to exhibit low light or bright lightcharacteristics as desired.

Exposure is the total amount of light allowed to fall on a sensor duringthe process of taking a photograph. Exposure control is control of thetotal amount of light incident on a sensor during the process of takinga photograph.

In contrast to exposure control, which is used by conventional digitalcameras to manage dynamic range, the digital camera systems of anembodiment use integration time control to control the time theelectrical signal is integrated on a charge storage device (capacitance)within a sensor (pixel), as described herein. Integration time control,also referred to as “focal plane shutter” control, controls the time theelectrical signal is integrated or accumulated by controlling a switch(e.g., charge integration switch) coupled or connected to the sensor ora photo-detection mechanism of a sensor. For example, the chargeintegration switch is placed in a state to allow charge to accumulatewithin the sensor for a period of time approximately equal to theintegration time corresponding to that sensor; upon completion of theintegration period, the switch is placed in a state to transfer theaccumulated charge as a photo-signal to a processing component. Digitalcamera components or circuitry are configured to allow independentcontrol of the charge integration switch associated with each sensor,thereby making possible dynamic range control for each sensor. Theintegration time control can be executed (depending on readoutconfiguration) according to a number of techniques, for example, rollingmode and/or snap-shot mode to name a few.

The output of the analog signal logic is supplied to the black levelcontrol, which determines the level of noise within the signal, andfilters out some or all of such noise. If the sensor coupled to thechannel processor is focused upon a narrower band of visible spectrumthan traditional image sensors, the black level control can be morefinely tuned to eliminate noise.

The output of the black level control is supplied to the exposurecontrol, which measures the overall volume of light being captured bythe array and adjusts the capture time for image quality. Traditionalcameras must make this determination on a global basis (for all colors).In the camera of an embodiment, however, the exposure control can bespecifically adapted to the wavelength (or band of wavelengths) to whichthe sensor is configured. Each channel processor is thus able to providea capture time that is specifically adapted to the sensor and/orspecific color (or band of colors) targeted, and which may be differentthan the capture time provided by another channel processor for anotheroptical channel.

FIG. 18C is a block diagram of the image pipeline, under an embodiment.In this embodiment, the image pipeline includes two portions. The firstportion includes a color plane integrator and an image adjustor. Thecolor plane integrator receives an output from each of the channelprocessors and integrates the multiple color planes into a single colorimage. The output of the color plane integrator, which is indicative ofthe single color image, is supplied to the image adjustor, which adjuststhe single color image for saturation, sharpness, intensity and hue. Theadjustor also adjusts the image to remove artifacts and any undesiredeffects related to bad pixels in the one or more color channels. Theoutput of the image adjustor is supplied to the second portion of thepipeline, which provides auto focus, zoom, windowing, pixel binning andcamera functions.

FIG. 18D is a block diagram of the image post processor, under anembodiment. In this embodiment, the image post processor includes anencoder and an output interface. The encoder receives the output signalfrom the image pipeline and provides encoding to supply an output signalin accordance with one or more standard protocols (e.g., MPEG and/orJPEG). The output of the encoder is supplied to the output interface,which provides encoding to supply an output signal in accordance with astandard output interface, e.g., universal serial bus (USB) interface.

FIG. 19 is a block diagram of digital camera system, including systemcontrol components, under an embodiment. The system control portionincludes a serial interface, configuration registers, power management,voltage regulation and control, timing and control, a camera controlinterface and a serial interface, but is not so limited. In someembodiments, the camera interface comprises an interface that processessignals that are in the form of high level language (HLL) instructions.In some embodiments the camera interface comprises an interface thatprocesses control signals that are in the form of low level language(LLL) instructions and/or of any other form now known or laterdeveloped. Some embodiments may process both HLL instructions and LLLinstructions.

As used herein, the following terms are interpreted as described below,unless the context requires a different interpretation.

“Array” means a group of photodetectors, also know as pixels, whichoperate in concert to create one image. The array captures photons andconverts the data to an electronic signal. The array outputs this rawdata to signal processing circuitry that generates the image sensorimage output.

“Digital Camera” means a single assembly that receives photons, convertsthem to electrical signals on a semiconductor device (“image sensor”),and processes those signals into an output that yields a photographicimage. The digital camera would included any necessary lenses, imagesensor, shutter, flash, signal processing circuitry, memory device, userinterface features, power supply and any mechanical structure (e.g.circuit board, housing, etc) to house these components. A digital cameramay be a stand-alone product or may be imbedded in other appliances,such as cell phones, computers or the myriad of other imaging platformsnow available or to be created in the future, such as those that becomefeasible as a result of this disclosure.

“Digital Camera Subsystem” (DCS) means a single assembly that receivesphotons, converts them to electrical signals on a semiconductor device(“image sensor”) and processes those signals into an output that yieldsa photographic image. The Digital Camera Subsystem includes anynecessary lenses, image sensor, signal processing circuitry, shutter,flash and any frame to hold the components as may be required. The powersupply, memory devices and any mechanical structure are not necessarilyincluded.

“Electronic media” means that images are captured, processed and storedelectronically as opposed to the use of film.

“Frame” or “thin plate” means the component of the DCS that is used tohold the lenses and mount to the image sensor.

“Image sensor” means the semiconductor device that includes the photondetectors (“pixels”), processing circuitry and output channels. Theinputs are the photons and the output is the image data.

“Lens” means a single lens or series of stacked lenses (a column oneabove the other) that shape light rays above an individual array. Whenmultiple stacks of lenses are employed over different arrays, they arecalled “lenses.”

“Package” means a case or frame that an image sensor (or anysemiconductor chip) is mounted in or on, which protects the imager andprovides a hermetic seal. “Packageless” refers to those semiconductorchips that can be mounted directly to a circuit board without need of apackage.

The terms “Photo-detector” and “pixels” mean an electronic device thatsenses and captures photons and converts them to electronic signals.These extremely small devices are used in large quantities (hundreds ofthousands to millions) in a matrix to capture an image.

“Semiconductor Chip” means a discrete electronic device fabricated on asilicon or similar substrate, which is commonly used in virtually allelectronic equipment.

“Signal Processing Circuitry” means the hardware and software within theimage sensor that translates the photon input information intoelectronic signals and ultimately into an image output signal.

Aspects of the digital camera systems and methods described herein maybe implemented as functionality programmed into any of a variety ofcircuitry, including programmable logic devices (PLDs), such as fieldprogrammable gate arrays (FPGAs), programmable array logic (PAL)devices, electrically programmable logic and memory devices and standardcell-based devices, as well as application specific integrated circuits(ASICs). Some other possibilities for implementing aspects of thedigital camera systems include: microcontrollers with memory (such aselectronically erasable programmable read only memory (EEPROM)),embedded microprocessors, firmware, software, etc. Furthermore, aspectsof the digital camera systems may be embodied in microprocessors havingsoftware-based circuit emulation, discrete logic (sequential andcombinatorial), custom devices, fuzzy (neural) logic, quantum devices,and hybrids of any of the above device types. Of course the underlyingdevice technologies may be provided in a variety of component types,e.g., metal-oxide semiconductor field-effect transistor (MOSFET)technologies like complementary metal-oxide semiconductor (CMOS),bipolar technologies like emitter-coupled logic (ECL), polymertechnologies (e.g., silicon-conjugated polymer and metal-conjugatedpolymer-metal structures), mixed analog and digital, etc.

The functions described herein can be performed by programs or sets ofprogram codes, including software, firmware, executable code orinstructions running on or otherwise being executed by one or moregeneral-purpose computers or processor-based systems. The computers orother processor-based systems may include one or more central processingunits for executing program code, volatile memory, such as RAM fortemporarily storing data and data structures during program execution,non-volatile memory, such as a hard disc drive or optical drive, forstoring programs and data, including databases and other data stores,and a network interface for accessing an intranet and/or the Internet.However, the digital camera systems and methods may also be implementedusing special purpose computers, wireless computers, state machines,and/or hardwired electronic circuits.

It should be noted that the various circuits disclosed herein may bedescribed using computer aided design tools and expressed (orrepresented), as data and/or instructions embodied in variouscomputer-readable media, in terms of their behavioral, registertransfer, logic component, transistor, layout geometries, and/or othercharacteristics. Formats of files and other objects in which suchcircuit expressions may be implemented include, but are not limited to,formats supporting behavioral languages such as C, Verilog, and HLDL,formats supporting register level description languages like RTL, andformats supporting geometry description languages such as GDSII, GDSIII,GDSIV, CIF, MEBES and any other suitable formats and languages.Computer-readable media in which such formatted data and/or instructionsmay be embodied include, but are not limited to, non-volatile storagemedia in various forms (e.g., optical, magnetic or semiconductor storagemedia) and carrier waves that may be used to transfer such formatteddata and/or instructions through wireless, optical, or wired signalingmedia or any combination thereof. Examples of transfers of suchformatted data and/or instructions by carrier waves include, but are notlimited to, transfers (uploads, downloads, e-mail, etc.) over theInternet and/or other computer networks via one or more data transferprotocols (e.g., HTTP, FTP, SMTP, etc.). When received within a computersystem via one or more computer-readable media, such data and/orinstruction-based expressions of the above described components may beprocessed by a processing entity (e.g., one or more processors) withinthe computer system in conjunction with execution of one or more othercomputer programs.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense as opposed to anexclusive or exhaustive sense; that is to say, in a sense of “including,but not limited to.” Words using the singular or plural number alsoinclude the plural or singular number respectively. Additionally, thewords “herein,” “hereunder,” “above,” “below,” and words of similarimport refer to this application as a whole and not to any particularportions of this application. When the word “or” is used in reference toa list of two or more items, that word covers all of the followinginterpretations of the word: any of the items in the list, all of theitems in the list and any combination of the items in the list.

The above description of illustrated embodiments of the digital camerasystems and methods is not intended to be exhaustive or to limit thedigital camera systems and methods to the precise form disclosed. Whilespecific embodiments of, and examples for, the digital camera systemsand methods are described herein for illustrative purposes, variousequivalent modifications are possible within the scope of the digitalcamera systems and methods, as those skilled in the relevant art willrecognize. The teachings of the digital camera systems and methodsprovided herein can be applied to other systems and methods, not onlyfor the systems and methods described above.

The elements and acts of the various embodiments described above can becombined to provide further embodiments. These and other changes can bemade to the digital camera systems and methods in light of the abovedetailed description.

In general, in the following claims, the terms used should not beconstrued to limit the digital camera systems and methods to thespecific embodiments disclosed in the specification and the claims, butshould be construed to include all systems that operate under theclaims. Accordingly, the digital camera systems and methods are notlimited by the disclosure, but instead the scope of the digital camerasystems and methods is to be determined entirely by the claims.

While certain aspects of the digital camera systems and methods arepresented below in certain claim forms, the inventors contemplate thevarious aspects of the digital camera systems and methods in any numberof claim forms. Accordingly, the inventors reserve the right to addadditional claims after filing the application to pursue such additionalclaim forms for other aspects of the digital camera systems and methods.

1. (canceled)
 2. A multi-lens camera arrangement comprising: a firstchannel including a first optics component and a first image sensor,wherein the first optics component generates visual spectrum data; asecond channel including a second optics component and a second imagesensor, wherein the second optics component generates infrared spectrumdata; and a processing component coupled to the first optics componentand the second optics component to combine outputs from the firstchannel and the second channel to generate a combined image.
 3. Themulti-lens camera arrangement of claim 1, wherein the visual spectrumdata includes RGB data.
 4. The multi-lens camera arrangement of claim 1,wherein the infrared spectrum data includes near infrared data.
 5. Themulti-lens camera arrangement of claim 1, wherein the first image sensorincludes a plurality of pixels.
 6. The multi-lens camera arrangement ofclaim 5, wherein each of the plurality of pixels includes two or morephotodiodes.
 7. The multi-lens camera arrangement of claim 6, wherein afirst photodiode detects a first band of colors and a second photodiodedetects a second band of colors.
 8. The multi-lens camera arrangement ofclaim 1, wherein the second image sensor detects low light signals. 9.The multi-lens camera arrangement of claim 1, wherein the first opticscomponent has a first field of view, and wherein the second opticscomponent has a second field of view that is different than the firstfield of view.
 10. The multi-lens camera arrangement of claim 1, whereinthe processing component simultaneously processes the visual spectrumdata and the infrared spectrum data.
 11. A camera system comprising: avisual spectrum camera having a visual spectrum sensor, wherein thevisual spectrum camera is configured to output visual spectrum data; aninfrared spectrum camera having an infrared spectrum sensor, wherein theinfrared spectrum camera is configured to output infrared spectrum data;a processor coupled to the visual spectrum camera and the infraredspectrum camera that combines the visual spectrum data and the infraredspectrum data.
 12. The camera system of claim 11, wherein the visualspectrum camera provides multiple images comprising a video output. 13.The camera system of claim 11, wherein the processor is configured todetermine a first integration time for the visual spectrum sensor and asecond integration time for the infrared spectrum sensor for a frame.14. The camera system of claim 11, wherein the processor is configuredto simultaneously control data acquisition by the visual spectrum sensorand the infrared spectrum sensor.
 15. The camera system of claim 11,wherein the processor combines the visual spectrum data and the infraredspectrum data without interpolation in a red-green-blue (RGB) colorspace.
 16. A tangible computer-readable medium having instructionsstored thereon, the instructions comprising: instructions to generatevisible spectrum data with a first channel of a digital camera, whereinthe first channel includes a first optics component and a first imagesensor; instructions to generate infrared spectrum data with a secondchannel of the digital camera, wherein the second channel includes asecond optics component and a second image sensor; and instructions togenerate an image by combining outputs of the first channel and thesecond channel.
 17. The tangible computer-readable medium of claim 16,wherein the first image sensor includes a plurality of pixels.
 18. Thetangible computer-readable medium of claim 16, wherein the visualspectrum data and the infrared spectrum data are processedsimultaneously.
 19. The tangible computer-readable medium of claim 16,further comprising instructions to simultaneously control dataacquisition by the first image sensor and the second image sensor. 20.The tangible computer-readable medium of claim 16, wherein the visiblespectrum data includes RGB data.
 21. The tangible computer-readablemedium of claim 16, wherein the infrared spectrum data includes nearinfrared data.